Knowledge distillation is used, in generative language modeling, to train a smaller student model using the help of a larger teacher model, resulting in improved capabilities for the student model. In this paper, we formulate a more general framework for knowledge distillation where the student learns from the teacher during training, and also learns to ask for the teacher's help at test-time following rules specifying test-time restrictions. Towards this, we first formulate knowledge distillation as an entropy-regularized value optimization problem. Adopting Path Consistency Learning to solve this, leads to a new knowledge distillation algorithm using on-policy and off-policy demonstrations. We extend this using constrained reinforcement learning to a framework that incorporates the use of the teacher model as a test-time reference, within constraints. In this situation, akin to a human learner, the model needs to learn not only the learning material, but also the relative difficulty of different sections to prioritize for seeking teacher help. We examine the efficacy of our method through experiments in translation and summarization tasks, observing trends in accuracy and teacher use, noting that our approach unlocks operating points not available to the popular Speculative Decoding approach.

Tim Ballaret once dreamed of becoming a stockbroker but ultimately found fulfillment helping high school students in south Los Angeles understand the relevance of math and science to their daily lives. But making engaging class materials is time-consuming, so this spring he started experimenting with generative AI tools. Recommendations by friends and influential teachers on social media led Ballaret to try MagicSchool, a tool for K-12 educators powered by OpenAI's text generation algorithms. He used it for tasks like creating math word problems that match his students' interests, like Taylor Swift and Minecraft, but the real test came when he used MagicSchool this summer to outline a year's worth of lesson plans for a new applied science and engineering class. "Taking back my summer helped me be more refreshed for a new school year," he says.